Image based rendering allows synthesis of a virtual viewpoint from a collection of camera images. For example, in an arrangement where a subject is surrounded by a ring of physical cameras, a new (virtual camera) view of the subject, corresponding to a position in between (physical camera) captured views, can be synthesised from the captured views or video streams if sufficient knowledge of the camera configuration and the scene captured by the physical cameras is available.
In recent times, the ability to synthesise an arbitrary viewpoint has been promoted for the purpose of “free viewpoint” video. In “free viewpoint” video, the viewer is able to actively adjust the camera viewpoint to his or her preference within constraints of the video capture system. Alternatively, a video producer or camera person may employ free viewpoint technology to construct a viewpoint for a passive broadcast audience. In the case of sport broadcast, a producer or camera person is tasked with constructing virtual camera viewpoints in an accurate and timely manner in order to capture the relevant viewpoint during live broadcast of the sport.
Industry standard methods of positioning virtual cameras in virtual environments, such as methods employed in 3D modelling software, used for product concept generation and rendering such as 3D Studio Max, Blender, and so forth, are known. In such systems, virtual cameras are configured by selecting, moving and dragging the virtual camera, a line of sight of the virtual camera, or both the virtual camera and the line of sight of the virtual camera. The movement of the camera may be constrained by changing the angle from which a three dimensional (3D) world is viewed, by using a 3D positioning widget or by activating constraints in the user interface (UI) (e.g. selecting an active plane). Clicking and dragging with a mouse to set both the camera position and line of sight (orientation) in the 3D environment is possible. However, editing other camera settings such as field of view or focal distance is done using user interface controls.
Methods of moving physical cameras in the real world such as remote control of cable cam and drone based cameras, or other robotic cameras, are known. The methods involving remote controls may be used to configure virtual cameras in real or virtual environments, such as for movie special effects sequences or console games. Configuring cable cam and drone cameras involves using one or more joysticks or other hardware controller to change the position and viewpoint of the cameras. The cable cam and drone systems can position cameras accurately but not quickly, as time is required to navigate the camera(s) into position. The delay caused by navigation makes the remote control systems less responsive to the action on a sports field, playing arena, or stadium which can often be fast-paced. Changing other camera settings such as zoom (field of view), focal distance (focus) is achieved by simultaneously manipulating other hardware controllers such as ‘zoom rockers’ or ‘focus wheels’. Manipulating the hardware controllers often requires two hands, sometimes two operators (four hands), and is time consuming.
Another known method of configuring virtual cameras involves navigating a space using virtual reality headgear, mobile phones or tablet computers, which present to the user an appearance of being present in a virtual scene. The virtual reality headgear, mobile phones or tablet computers can respond to motions of the user so as to maintain the illusion of looking into a virtual space, by matching motions of the device to compensatory changes to the position and orientation of the virtual camera in the virtual scene. Typically, the virtual camera motions are matched to the motions of the user using inertial navigation methods such accelerometers, tracking the surrounds using a camera, or using a GPS unit to locate the user in a large space such as a city.
The camera control interactions described above are typically inappropriate for applications, such as sport broadcast, as camera navigation using the interaction and systems described above is relatively time consuming or unable to follow fast action in real time. There remains an unmet need in virtual camera control for a method of generating and controlling a virtual camera view in an accurate and timely manner.